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ABSTRACT:  The  theoretical  basis  of  a  method  of  computing  the  depth  image  from 
the  ratio  of  two  intensity  images  is  described.  One  implementation  of  this  method 
forms  a  512x512  pixel  depth  image  in  30  seconds,  25  of  which  are  used  to  digitize 
images,  with  an  average  deviation  of  about  0.6%  of  the  depth  of  the  workspace,  a 
resolution  which  is  sufficient  for  experiments  in  shape  analysis  and  object  recogni- 
tion. This  report  is  written  to  save  as  an  introduction  of  the  method  to  new  users 
of  the  depth  sensor. 


Introduction: 

The  spatial  coordinates  (x,y,z)  of  a  point  on  the  surface  of  an  object  can  be  obtained  from 
the  intersection  of  the  appropriate  ray  frOTi  a  camera  (indexed  e.g.  by  the  pixel  location  in  the 
image)  with  a  vertical  plane  from  a  projector  which  is  displaced  boiizontaQy  from  the  camera. 
Fig.  1  sketches  a  two  dimensional  reduction  of  this  process.  A  number  of  plane  of  light  triangula- 
tion  schemes  have  been  implemented,  starting  in  the  early  1970's  [1,2]  and  continuing  through 
recent  commercial  products  [3,4]. 

Data  from  plane  of  light  sensors  is  usuaQy  analyzed  in  its  natiu^  form  of  lines  (from  the 
intersection  of  the  planes  of  light  with  objects)  [5-7].  For  example,  planar  object  surfaces  show 
parallel,  evenly  spaced  straight  line  segments,  the  ends  of  line  segments  denote  edges  which  can  be 
fit  by  straight  lines  or  curves,  etc.  From  this  information  features  of  interest  (holes,  edges)  or 
entire  objects  can  be  identified.  However,  data  for  an  entire  scene  can  only  be  built  up  from  a 
number  of  sequentially  processed  images  of  planes  of  light. 

Recently  a  method  of  acquiring  depth  information  from  an  entire  scene  at  once  was  pro- 
posed [8]  which  uses  the  ratio  of  two  images  formed  with  the  work  area  illuminated  sequentially 
by  two  beams  of  light  having  different  btensity  variations  across  them.  The  resulting  "ratio 
image"  provides  a  continiujus  coding  of  the  projected  beam  which  allows  the  identification  of  an 
arbitrarily  large  number  of  planes  of  light  in  the  workspace.  Experiments  with  a  preliminary 
implementation  of  this  method  [9]  established  that  the  assumptions  on  which  the  ratio  image  depth 
sensor  is  based  (reproducible  lighting  conditions,  linear  response  of  the  camera  to  light)  could  be 
achieved  technically.  The  actual  computation  of  depth  values  in  that  preliminary  work  was  done 
by  linear  interpolation  between  ratio  images  of  a  plane  placed  first  at  the  front  of  the  workspace, 
then  at  the  back,  a  procedure  which  produced  depth  images  characterized  by  systematic  distortion 
from  the  inappropriate  use  of  linear  interpolation  and  by  low  resolution  in  depth  due  to  magnify- 
ing the  small  differences  of  large  numbers.  A  quadratic  interpolation  method  of  calculation  using 
three  calibration  planes  reduced  systematic  distortions  but  at  a  cost  of  increased  "noise"  in  the 
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depth  images. 

An  observation  by  Robert  Hummel  led  to  the  development  of  the  present  look-up  table 
method  of  computing  the  depth  image  of  a  work  scene  from  the  ratio  image  of  the  scene.  Since 
the  depth  image  is  merely  the  ratio  image  run  through  a  look-up  table  plus  smaD  corrections,  the 
resolution  of  the  depth  image  is  the  same  as  that  of  the  ratio  image,  which  can  be  8  or  9  bits.  The 
method  produces  depth  images  entirely  free  of  systematic  distortions.  Since  look-up  tables  can  be 
loaded  into  the  VICOM,  all  processing  except  the  off-line  preparation  of  the  look-up  table  is  done 
quickly  on  this  machine. 

Section  2  of  this  report  details  the  derivation  of  the  look-up  table  method.  Section  3 
describes  the  implementation  of  the  method  on  the  VIODM  Section  4  describes  calibration 
methods  used  to  gather  data  necessary  to  construct  the  ratio-to-depth  look-up  table.  Section  5 
describes  in  broad  terms  how  the  look-up  table  is  calculated  in  the  present  implementation;  the 
actual  code  (in  C)  is  contained  in  Appendix  A.  Section  6  contains  images  formwd  using  this  look- 
up table  method  and  a  short  discussion  of  the  present  application  of  the  method. 

2.  Mathematical  Basis  of  the  Look-up  Table  Method 

Figure  2  sketches  the  geometrical  relations  which  underlie  the  look-up  table  method.  We 
consider  for  simplicity  the  two  dimensional  case,  which  extends  trivially  to  three  dimensions  when 
vertical  planes  of  light  are  used  m  place  of  the  2D  rays  from  the  projector.  The  camera  lies  at 
the  origin  of  the  coordinate  system  and  looks  along  the  z-axis,  and  rays  from  the  projector  diverge 
from  the  point  (xq,Z(^.  An  arbitrary  point  in  the  workspace  (x^)  is  determined  by  the  intersection 
of  the  ray  from  the  camera  at  angle  9  to  the  z-oxis  and  the  ray  from  the  projector  which  intersects 
the  z-axis  at  the  point  (0,d). 

For  convenience  in  the  following  argument  we  draw  line  segments  from  (x^)  and  (xq^q)  per- 
pendicular to  the  z-axis.    Then  by  similar  triangles  it  is  dear  that 
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In  this  expression  the  quantities  are  considered  to  be  coordinate  values,  and  thus  can  be  negative, 
as  e.g.  Xq  is  here.  This  can  be  rewritten  as 

X X  =  d 


or 
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x[l  -  ^2lli]  =  i 

Xq       Z 

or, 

d 

z  = 

where  xJz  has  been  written  as  tan6. 

The  key  observation  at  this  point  is  that  if  the  ratio  varies  strictly  monotonkally  across  the 
beam,  every  ratio  value  corresponds  to  a  unique  value  of  <i  on  the  z-axis.  Thus  within  the  beam 
it  is  possible  to  defme  a  function  </(p)  which  maps  the  observed  ratio  value  p  into  a  distance  on 
the  z-axis.  For  regions  of  the  image  sufficiently  dose  to  the  center  vertical  line,  tan6  is  very 
small,  and  this  d{p)  function  (which  is  easily  implemented  as  a  look-up  table)  converts  the  ratio 
observed  at  any  pixel  to  the  depth   (z  coordinate)  of  the  part  of  an  object  imaged  at  that  pixel. 


For  purposes  of  translating  ratio  images  to  depth  images  we  can  simply  replace  the  </  of  eq.  2  with 

d(p). 


3.  The  VICDM  Implementation 

The  VICOM  [10]  is  a  special  purpose  computer  designed  for  image  processing.  It  has  a 
firmware  operating  system  to  implement  low  level  commands,  and  serial  and  parallel  connections 
to  a  host  computer  (a  VAX  750  running  Unix  4.2  bsd)  to  allow  the  passing  of  commands  and 
images  between  the  machines.  The  shell  vsh  [11]  provide  a  very  convenient  softMffitte  interface. 

The  basic  data  unit  of  the  VICOM  is  the  512x512x12  bit  memory  plane.  Programs  written  to 
run  on  the  VICOM  (chain  files  of  VICCM  commands,  or  "vsh  macros")  can  be  thought  of  as 
being  written  in  a  marine  language  with  these  memory  planes  as  registers;  individual  pixels  can- 
not be  addressed  without  a  great  deal  of  effort.  The  baisic  processes  of  addition,  subtraction,  mul- 
tiplication, and  division  work  on  corresponding  pixels  of  images:  Ajj  +  Bjj  =  Dij  for  all  <  J  in 
the  images  A,  B,  and  D.  In  addition  there  is  the  ability  to  run  an  image  through  a  look-up  table, 
and  there  are  numerous  functions  useful  in  image  processing.  The  availability  of  only  12  bits  dur- 
ing all  processing  can  limit  attainable  depth  resolution  unless  special  effort  is  made  in  writing  com- 
mand files. 

To  improve  depth  resolution  eq.  2  should  be  written  so  the  available  bits  span  the  workspace 
exactly.  For  example,  suppose  that  all  depths  are  to  be  measured  from  a  plane  perpendicular  to 
the  z-axis  at  do,  and  let  z  =  Z  -(-  dg  relate  the  new  depths  Z  to  the  old  depths  z.  Making  this  sub- 
stitution in  eq.  2  results  in 

,    8(p)  +  JqC  ,. 
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Zq- 8(p) 
where  B(p)  is  now  the  look-q)  table,  and  C  = tan6    is  a  small  correction.    In  this  Z 

representation  all  2047  positive  numbers  (from  0  to  1.0  on  the  VICOM)  can  be  used  to  represent 
depths  within  the  imaged  area.  This  places  a  limit  of  resolution  of  about  0.5  mm  for  a  work  area 
of  1  meter  extent,  beyond  the  present  resolution  of  the  ratio  image.  The  depth  values  generated 
by  the  sensor  are  converted  to  physical  dimensions  by  multiplying  by  the  "con^ter  unit",  wiiich  is 
the  physical  distance  corresponding  to  1.0  in  the  VICOM  units.  The  con^nrter  unit  is  created  by 
the  program  which  makes  the  look-up  table. 

Since  all  processing  must  be  performed  on  an  image  as  a  whole,  it  may  not  be  immediately 
dear  how  eq.  3  can  be  implemented,  since  tand  is  spatiaDy  varying.  However,  one  need  merely 
create  an  image  for  each  quantity  on  the  right  hand  side  of  eq.  3.  The  look-up  table  can  be  used 
to  pass  constants  such  as  d^  and  xq  to  the  program  miming  on  the  VICOM.  However,  the  correc- 
tion C  must  be  approadied  carefully,  since  Zq  and  xq  are  both  larger  than  a  computer  unit,  and 

Zo 
thus  do  not  fit  in  the  VICOM  memory.   C  is  formed  by  combining  the  quantities  — ,  I/tq.  and 

Jq,  each  of  which  is  constrained  to  be  smaller  than  one  when  the  look-up  table  is  made.  These 
values  are  passed  in  the  ratio-to-depth  look-up  table  using  ratio  indices  outside  the  normal  range. 

To  make  a  depth  image  it  is  necessary  to  make  caUbration  measurements,  create  the  look-up 

table,  and  acquire  a  suitable  ratio  image  of  the  scene.  The  ratio  image  is  nm  through  the  look-up 

table  to  give  the  raw  depth  image.  Three  constant  images  are  made  with  appropriate  index  values 

Zo 
and  nm  through  the  look-up  table  to  give  — ,  1/xq,  and  d^  images.  The  tan6  image  must  m  fact 

by  created  by  the  host  computer  (in  the  present  experiments  a  straight  line  approximation  is 
used),  but  can  be  passed  as  a  small  16x512  image  to  the  VICC^  and  stored  in  the  same  large 


plane  as  the  look-up  table.  This  small  image  is  copied  several  times  to  form  a  full  size  image. 
Another  image  is  made  for  the  constant  1.  At  this  point  each  quantity  of  Eq.  3  exists  as  an 
image,  and  VICOM  commands  can  be  used  to  compute  the  fully  correct  depth  image. 


4.  Calibration  Procedures 

The  information  necessary  to  construct  the  look-up  table  is  simply  the  value  of  the  ratio  at 
the  z-axis  where  it  is  illuminated  by  the  projector.  K  the  projector  optics  and  the  transmissivity  of 
the  filters  were  known,  then  the  d(p)  function  could  be  calculated  for  any  desired  geometry.  It  is 
easier,  however,  to  measure  the  projeaed  ratios  in  a  way  wfaicfa  is  more  directly  applicable  to  the 
calibration  procedure. 

A  direct  measurement  of  d{p)  is  difficult  since,  as  shown  in  Fig.  3,  the  paints  oif  interest  are 
all  imaged  on  the  same  point  of  the  two  dimensional  camera,  or  to  a  vertical  line  in  three  dimen- 
sions. However,  the  projected  ratios  can  be  determined  by  moving  the  screen  to  a  more  con- 
venient angle  to  the  camera.  Measurements  would  be  made  (perhaps  with  the  screen  in  several 
different  locations)  and  the  ratios  observed  on  the  screen  could  be  computationally  mapped  to  give 
d(p)  on  the  z-axis. 

The  method  which  has  been  used  in  generating  the  look-up  tables  for  the  present  experi- 
ments in  fact  does  not  remap  the  ratio  values.  The  configuration  shown  in  Fig.  3  is  altered  by 
rotating  the  projector  and  the  screen  90°  about  the  central  point  of  the  workspiaoe,  defined  as  the 
intersection  of  the  central  ray  from  the  camera  and  the  central  ray  from  the  projector.  The  pro- 
jector must  retain  the  same  angle  to  the  screen  in  this  calibration  set-up  as  to  the  central  camera 
ray  in  the  measuring  geometry.  In  this  way  512  ratio  values  corresponding  to  the  512  horizontal 
pixels  are  measured  at  once.  Additional  tolerance  to  local  ratio  deviations,  due  perhaps  to  dirt  on 
the  filters  or  in  the  optics,  is  obtained  by  averaging  vertical  columns,  since  each  column  should 
have  the  same  ratio  value.  In  the  present  implementation  of  this  method  16  rows  spaced  along 
the  central  half  of  the  image  are  copied  to  the  VAX  for  averaging  and  forming  the  look-up  table. 

One  difficulty  with  this  method  is  that  some  areas  which  intuitively  should  be  part  of  the 
working  area  of  the  depth  sensor  lie  outside  the  region  illuminated  by  the  projector.  Fig.  4  illus- 
trates the  problem:  the  shaded  comers  of  the  square  are  not  part  of  the  workiiig  area.  This  prob- 
lem is  overcome  by  making  the  calibration  measurements  with  the  projector  at  a  distaiice  chosen 
so  the  beam  conveniently  fills  the  field  of  view  of  the  camera,  tlicn  setting  up  for  depth  measure- 
ments with  the  projector  at  a  greater  distance  or  with  the  zoom  lens  adjusted  to  give  a  larger 
beam.  The  calibration  process  thus  consists  of  two  procedures;  first,  the  ratio  is  recorded  with  the 
projector  at  the  chosen  angle  to  the  screen  (equipment  in  the  calibration  arrangement),  and 
second,  the  coordinates  of  the  projector  and  the  distance  to  the  center  of  the  workspace  (camera  is 
at  the  origin)  are  recorded  (with  equipment  in  the  measuring  location).  A  full  description  of  the 
process  follows: 

1)  Place  all  apparatus  in  the  measuring  position.  Bear  in  mind  that  depth  acxuracy 
increases  and  quality  of  illumination  decreases  as  the  angle  between  the  projector 
beam  and  the  camera  axis  approaches  90°;  45°  has  worked  weO  as  a  compromise. 

2)  Use  the  "CAM"  command  to  place  a  screen  along  the  camera  axis,  defined  by  a 
vertical  line  on  the  center  of  the  monitor.  A  screen  of  matte  white  Formica  mounted 
on  heavy  plywood  combines  flatness  and  good  properties  of  reflection. 

3)  Using  a  slide  with  nuirkings,  find  the  center  of  the  projector  beam  on  this  screen. 
Record  the  angle  of  the  projector  to  the  screen. 

4)  Rotate  the  screen  90°  so  it  is  square  to  the  camera  (use  perpendicularity  device). 

5)  Place  the  projector  at  the  same  angle  to  the  screen  as  in  step  3  and  at  such  a  dis- 
tance that  the  useful  portion  of  the  beam  fills  the  field  of  view  of  the  camera.  Adjust 
the  equipment  so  the  center  of  the  beam  falls  on  the  screen  at  the  center  of  the  cam- 
era field  of  view.  If  the  angle  of  the  projector  to  the  screen  is  changed,  record  the 
new  angle  for  use  in  setting  up  the  projector  for  measurements.   Adjust  the  projector 


so  the  projected  slides  are  aligned  with  tbe  vertical. 

6)  In  this  calibration  geometry,  record  the  width  of  index  marks  projected  on  the 
screen  using  a  specially  prepared  slide.  A  grid  slide  is  convenient  for  this  and  for 
adjusting  the  projector  focus.  (In  these  measurements  the  projector  and  camera  have 
been  focussed  on  the  center  of  the  workspace). 

7)  Make  a  ratio  image  using  the  chosen  filters,  and  copy  16  lines  of  tbe  image  to  the 
VAX  as  "raw.img"  for  processing.  Multiple  digitizations  should  be  made  and  aver- 
aged to  reduce  the  effects  of  digization  noise  and  temporal  instability  of  the  projec- 
tor. Typically  8  successive  digitizations  are  made,  which  results  in  intensity  images 
having  about  10  good  bits.  Check  tbe  ratio  image  to  make  sure  the  lines  of  constant 
ratio  are  vertical  and  well  defined. 

8)  Place  the  projector  in  the  depth  measuring  position.  At  this  time  the  work  space 
is  defmed  by  the  placement  of  tbe  projector  and  tbe  size  of  tbe  projector  beam 
(which  can  be  varied  using  the  zoom  adjustment).  The  only  constraint  is  that  the 
angle  between  the  center  of  the  projector  beam  and  the  camera  axis  be  the  same  as 
the  angle  between  the  plane  of  the  screen  and  the  projector  beam  in  tbe  calibration 
geometry.  Check  that  planes  of  constant  ratio  are  vertical.  With  tbe  screen  along 
the  camera  axis  record  the  width  between  the  projected  index  marks  as  in  step  6. 
Record  the  distance  to  the  center  of  the  workspace  from  the  camera  and  tiie  x  and  z 
coordinates  of  the  projector.  (Unlike  the  pinhole  camera  and  projector  drawn  in  the 
figures,  real  optical  systems  have  no  single  point  through  which  all  rays  pass.  Exper- 
iments to  determine  the  effective  point  of  divergence  have  placed  this  point  near  the 
front  element  of  the  lens  systems,  and  nearly  any  point  used  consistently  seems  to 
work.) 

9)  Run  delcalc*,  the  program  which  makes  the  look-up  table.  Eater  the  projector 
location  and  the  distance  to  the  center  of  the  workspace.  When  prompted  for  the 
"grid  width  ratio"  enter  MW/CW,  wbere  MW  and  CW  are  the  widths  between  calibra- 
tion marks  observed  in  the  measuring  and  calibration  positions  respectively. 

At  this  point  the  VAX  creates  the  file  ddta.lut  which  is  used  in  the  chain  files  to  make  depth 
measurements  using  a  pair  of  filters.  This  file  can  be  used  to  make  depth  images  at  any  time  in 
the  future  by  using  the  same  filters  and  placing  the  projector  relative  to  the  camera  exactly  as  it 
was  when  the  look-up  table  was  made.  Note  that  this  method  results  in  a  work  area  which  is  sym- 
metrical in  depth  about  the  midpoint  of  tbe  beam,  d.e.  some  usable  ratio  and  dqjtb  pairs  have  not 
been  incorporated  into  the  look-up  table. 


5.  Overview  of  the  Program  delcalcc 

Commented  code  for  the  program  which  creates  the  look-up  table  is  reproduced  in  Appendix 
A    The  code  has  the  following  main  functions: 

Input.  This  prompts  for  the  projector  position,  distance  to  tbe  screen  and  tlie  grid  width 
ratio.  The  last  two  are  needed  to  construct  the  look-up  table;  the  projector  position  must  be  input 
to  ensure  that  later  computations  on  the  VIO^l  will  not  overflow.  The  distances  can  be  meas- 
ured using  any  units,  but  centimeters  are  a  good  choice. 

Computer  unit.  The  distance  corresponding  to  the  differraoe  between  the  dosest  and 
farthest  points  on  the  z  axis  which  can  be  recorded  with  tbe  equipment  in  measuring  position  is 
calculated.  Since  this  distance  is  to  be  mapped  into  depth  values  between  0  and  1,  this  distance  is 
the  computer  unit.  However,  if  the  distamce  from  tbe  camera  to  the  cento*  of  the  workspace  is 
larger  than  this  the  computer  unit  is  made  larger  (and  tbe  range  of  depdi  values  becomes 
correspondingly  smaller).  If  tbe  chosen  measuring  geometry  is  ina^yropriate  for  tbe  calculations 
an  error  message  is  generated  and  tbe  program  exits. 
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Averaging.  The  first  processing  of  the  raw  ratio  values,  indexed  by  horizontal  pixel  number, 
is  to  average  all  16  values  for  each  horizontal  position.  This  reduces  the  effects  of  noise  and 
makes  the  look-up  table  representative  of  a  larger  region  of  the  image. 

Smoothing.  Even  with  averaging  the  possibility  exists  that  noise  will  result  in  locally  non- 
monotonic behavior  in  the  averaged  points.  To  eiKourage  monotonic  ratios  this  routine  places  at 
each  point  the  average  value  of  the  7  ad)oimng  pixels  centered  on  that  point.  While  this  process 
does  not  guarantee  a  monotonic  ratio  sequence,  non-monotonic  behavior  has  not  yet  been 
observed. 

Inversion.  The  processing  so  far  has  dealt  with  ratio  values  indexed  by  pixel  location,  which 
corresponds  to  the  depth.  The  maketable  routine  turns  the  relation  around,  and  creates  a  file  of 
depth  values  indexed  by  ratio.  512  evenly  spaced  depth  values  are  computed  between  the  far  and 
near  points,  and  each  depth  value  is  placed  in  the  slot  of  its  correspoiiding  ratio.  SmaU  distortions 
which  exist  in  the  camera  optics  and  detector  have  not  been  taken  into  aaxnint  in  this  routine;  this 
results  in  a  systematic  distortion  on  the  order  of  0.5%  of  the  depth  of  the  workspace  for  extreme 
depths. 

Interpolation.  At  this  point  there  are  only  512  ratio  indices  filled  of  the  2047  positive  indices 
available,  so  the  unfilled  slots  are  filled  by  linear  interpolation  between  the  nearest  filled  neigh- 
bors. Depth  values  indexed  by  ratios  less  than  or  greater  than  the  filled  areas  are  set  to  zero. 

Parameter  passing.  The  locations  of  the  projector  and  the  center  of  the  work  space  are 
passed  in  a  form  which  can  be  used  in  the  VICOM  calculations,  namely  as  Z^x^,  d^,  and  Vx^  with 
the  values  in  computer  units.  These  are  passed  in  table  locations  4090,  4091,  and  4092  (FFA, 
FFB,  and  FFC  hex).  For  reference,  the  number  of  centimeters  in  a  computer  unit  is  passed,  mul- 
tiplied by  10  to  preserve  a  decimal  place,  in  4093  (FFD  hex). 

Writing.  When  complete,  the  table  of  4096  short  integers  is  written  to  disk  in  a  format 
which  will  be  loaded  correctly  by  the  vsh  interfacing  routines.  The  fDe  is  called  delta.lut,  and 
overwrites  any  previous  file  of  that  name. 


6.  Representative  Results 

To  illustrate  the  capabilities  of  this  implementation  of  the  ratio  image  depth  sensor  a  simple 
'work  scene'  was  constructed  having  a  sphere  and  letters  cut  from  paper  at  different  distances 
from  a  back  plane.  Fig.  5  is  an  intensity  image  of  this  scene  as  viewed  by  the  tv  camera.  Some 
notable  features  of  this  intensity  image  are  the  dark  paper  taped  to  the  white  screen  at  the  back  of 
the  scene,  the  grain  in  the  wood  block  at  the  lower  left,  and  the  way  the  letters  blend  into  the 
background.  (The  shadows  suggest  to  the  human  viewer  how  the  projector  is  offset  from  the  cam- 
era.) 

Fig.  6  is  a  photograph  of  the  depth  image  of  this  scene,  a  grey  scale  image  where  larger 
numbers  (brighter)  are  farther  from  the  camera.  The  bade  screen  is  uniformly  bright  because  the 
dark  paper  is  at  the  same  depth  as  the  white  screen.  Similarly  the  dark  grain  of  the  wood  block  is 
not  visible  in  the  depth  image.  The  letters  DEPTH  advance  towards  the  camera.  The  shadow 
regions  are  tagged  black  to  identify  them;  there  is  of  course  no  information  available  to  the  depth 
sensor  in  shadow  regions.  This  depth  image  was  formed  in  30  seconds,  of  which  25  seconds  were 
used  in  acquiring  the  ratio  image  (8  digitizations  of  each  intensity  image  were  made  to  reduce 
noise  by  averaging)  and  5  seconds  were  needed  for  computation. 

By  threshholding  the  depth  image  regions  of  the  image  having  a  narrow  range  of  values  can 
be  selectively  displayed.  In  Fig.  7,  for  example,  several  slices  of  the  depth  values  are  reproduced. 
Each  of  these  slices  is  about  two  cm  thick.  No  attempt  was  made  to  smooth  the  values  of  the  sur- 
faces by  spatial  averaging.  The  P  and  T  differ  by  about  5  cm  in  depth.  The  overaD  depth  of 
imaged  area  was  about  80  cm. 


Unwanted  variation  (noise)  in  the  intensity  images  translates  into  error  in  the  derived  depth 
images.  Analysis  of  the  error  in  these  depth  images  and  in  others  made  with  the  same  technique 
showed  a  random  variation  which  resulted  in  average  deviations  for  groups  of  6  pixels  of  between 
0.2  and  0.5  cm  (average  0.3  on)  in  a  bright  region,  and  between  0.45  and  1.05  cm  (average  0.8 
cm)  in  a  region  having  four  times  smallo:  reflectivity.  These  results  are  consistent  with  earlier 
uncertainty  estimates  [9].  There  was  also  a  coherent  variation,  where  the  entire  image  became 
closer  or  farther,  having  an  average  deviation  of  similar  size;  the  effect  is  probably  related  to  smaU 
temporal  variations  in  the  intensity  of  the  slide  projector  lamp.  Even  with  this  noise  the  relative 
change  of  location  of  a  dark  surface  by  0.3  cm  was  dearly  observed  when  4  adjacent  pixels  of  a 
subsampled  image  were  averaged.  The  ptxel-to-pixel  variation  in  depth  makes  a  surface  appear 
very  uneven  at  small  scales,  and  thus  gradient  operators  of  the  sort  often  used  for  edge  detectors 
(3x3  convolution  boxes)  give  very  noisy  results  on  depth  surfaces.  Jump  edges  of  objects  (where 
an  object  is  separated  from  the  background)  are  of  course  very  sharply  determined  by  edge  detec- 
tors. 

Besides  the  deviations  due  to  noise  and  intensity  variation  there  are  small  systematic  distur- 
bances to  the  depth  image.  The  most  important  of  these  is  that  of  secondary  illumination,  where 
light  is  reflected  from  an  object  in  the  beam  to  a  visible  surface.  Tnttatv^'<  of  this  are  visible  in 
Fig.  6,  both  in  the  scattered  points  in  the  shadow  of  the  P,  and  also  on  the  upper  right  limb  of  the 
sphere,  which  should  be  in  shadow.  The  e^ect  is  usually  very  small,  except  where  the  direct 
illumination  is  very  weak  or  nonexistent. 


7.  Summary 

The  present  implementation  of  the  depth  sensor  acquires  a  512x512  depth  image  in  30 
seconds,  25  of  which  are  used  to  acquire  the  ratio  image.  The  result  is  a  depth  image  with  an  aver- 
age deviation  of  approximately  ±  0.5  cm  over  a  working  area  of  80  an  centered  80  cm  from  the 
camera.  The  projector  is  70  cm  behind  and  150  cm  to  tiw  left  of  the  camera  in  these  experiments. 
The  calibration  measurements  take  about  15  minutes,  most  of  the  time  being  spent  in  setting  up 
and  making  geometrical  measurements.  Qnstruction  of  the  look-up  table  takes  about  30  seconds 
(elapsed  time,  including  data  input)  on  the  VAX. 

This  implementation  of  the  depth  sensor  provides  relatively  good  depth  data  quickly  and 
easily,  and  is  adequate  for  use  in  experiments  in  shape  analysis  and  object  recognition,  since  addi- 
tional accuracy  in  fitting  surfaces  to  depth  data  can  be  obtained  by  limited  spatial  avnaging.  In 
such  studies  the  slight  absolute  displacements  caused  by  the  intensity  fluctuations  should  not 
matter.  If  the  absolute  position  of  an  object  is  important,  as  for  instance  when  object  coordinates 
are  to  be  jjassed  to  a  robot,  these  fluctations  can  be  corrected  by  measuring  depth  relative  to  an 
object  in  a  known  location  in  the  workspace.  The  look-up  table  odculations  wfaidi  are  used  in  this 
implementation  are  also  used  in  the  sawtooth  filter  implementation  which  is  currently  under 
development. 


References: 

1.  Y.  Shirai  and  M.  Suwa,  Recognition  of  Polyhedrons  with  a  Range  Finder,  Proc  2nd  UCAI,  pp. 
80-87  (1971). 

2.  G.  J.  Agin  and  T.  O.  Binford,  Computer  Description  of  Curved  Objects,  Proc.  3rd  UCAI,  pp. 
629-640  (1973). 

3.  The  White  Scanner,  Technical  Arts  Corp.,  PO  Box  15390,  Seattle,  Washington  98115. 

4.  Robotic  Vision  Systems  Inc.,  425  Rabro  Dr.  East,  Hauppage  NY  11788. 

5.  K.  Sugihara,  Range  Data  Analysis  Guided  by  a  Junction  Dictionary,  Artificial  Intelligence  12  , 
pp.  41-69  (1979). 

6.  M.  Oshima  and  Y  Shirai,  Object  Recognition  using  3  Dimensional  Information,  Proc.  7th 
UCAI,  pp.  601-606  (1981). 


7.  R.  C.  BoUes,  P.  Hbraud  and  M  J.  Hannah,  3DPO:  Three-Dimensional  Parts  Orientation  Sys- 
tem, Proc.  8th  UCAI.  pp.  1116-1120  (1983). 

8.  J.  T.  Schwartz,  Stmctured  Light  Sensors  for  3-D  Robot  Vision,  NYU  Courant  Institute  Robot- 
ics Rpt.  no.  8  (1983). 

9.  C.  M  Bastu5check  and  J.  T.  Schwartz,  Preliminary  Implementation  of  a  Ratio  Image  Depth 
Sensor,  NYU  Courant  Institute  Robotics  Rpt.  no.  28  (1984). 

10.  Vicom  Systems,  Inc.,  2307  Bering  Drive,  San  Jose  CA  95131. 

11.  D.  Qark  and  R.  ILiramel,  VSH  Users'  Manual:  An  Image  Processing  Environment,  NYU 
Courant  Institute  Robotics  Tech.  Report  (Oct.  1984). 


Bi 
O 
U 


o 

H 


4J 

8 


o    !■ 


0 


I  5  ..  .. 

v"  *J    ff  i 


S|' 

II    u   • 
■  "0   u 

4J  ■ 

CUT} 

5  a.  II  ■ 
Slo 

iJ  u  V 


1^ 


■a  Tj 

>. 

8.1 

o 

•H  C 

«J  O 

•H  -H 

■O  *J 

C  a 
o  u 

"a 

5-3 


55 


1 


c  c 

o  o 

4J    tJ 

51^ 


g-    9 


■0 


^     o 

o      ^ 
o 

—  *J 


:  r1 

5. 


6%     ^ 

O         k.    o 

o  • 

"•a 


o  « 


O 

o 

4J         — 


r1 


O'         -5   *! 

X        u  o 


3        ST 

r 


3 

« 

i 


I  I 


•  : 
-t  C 


5  5- 
%1' 


r  9 


4J 

o. 


b2 


.'5 


II  II 


55|l 

u  u  8  I 
a.  a,*>  I 


4< 

8. 


8. 


~  5  5  f  1  I 


ik 


t    5 

■       o 

t  S 


1 

5 


m  o 


o  o 


—  g 


:§ 
:$ 


«<  M  v    —^ 

111  " 

S  o  o 

HB    X   M 

II    II    II 


o 

s 


I 


o 

H 
10 


'5       * 


n  a 
u  -o 


3 

'^ 

S 

-H 

■H 

a 
X 


o 

u 
D*  O 

■^" 

»  u 
a   3 


O 

5, 

«j  a 

o5 


-:5 


CD  ~i 


s^ 


*u  «  .-4  «  q  V 

o       •  M  tl 

■u  V  *>  o  > 

-I         X  X 

■g  ia  a~.  e 

«       3  ^  j=  o 

•HO'  3  . 


■3  •  o 


3 

.a 


i; 


a 

>. 

•  c 

3 

«l    O 

• 

ca  oi 


1^8 

lolS 

:  V  Ti  u 

a  c 

K  o  -*- 

_     o  rH  e    . 

■    X  U    X 

0  •  u  "--  e       0 

.    rH     O    O    i     U    ^ 

u  j:>  «  N  0 

rH  S  g  o  •»-  a 

a  a  8  «  V  Em 

0  a  a  s  -  Em 

■u  o  0  o  a 

3  0.9  a  ■->  u   c 

1-i  X;    rH    -H    -H    -4 

0   a  <"  -o  V 

0  -<  a 
■Hue    -  c 


§a 


C  4J  '^ 


> 


a 

•    3 


OtiM  X  u   • 


a  <  » 


TJ 


c  <l  o 

•n  ij  c  ■o  -H  •" 

i  -H  'I  -  a  o 

9  k.  a  -I  8 

e  J  a  .  T)  -1 

^  'O  u  u  a 

u  o  o  ~  ■ 


I      2 


«     O  '^ 


>      ©a. 


^2 


4^ 

3    3 


o 

o 

o 

\ 

o 

• 

a 

o 

^ 

o 

«4 

«4 

o 

i 

d 

*-l 

d 

m 

o 

^ 

IH 

d 

0» 

d 

• 

SK 

i 

I  83 

5  3  '- 

V  £  m 

a  4]  *< 


o 


^6. 

%  r-l     I 


•       ••       a       li  M       gsrf«<«4 


I 


10 


.    5 
I    9 


g 
3 


I 


5     I  • 


0. 


e   tl 


8. 

■ 

a, 

0 

3 


«]  o 

e 

•  a 
0  u 

^5 

0.  o 

a 

13 

S" 

3 


3    > 

• 

31 
1« 

a  a 
>- 

-I  o 

• 

^  « 

o  o 

•^  a 

a, 
m 
• 

O   3 

is 


8 


5 

o 


I 


I 


..  s 


o  — 

■  a 
-4  \  3 


:i    -^ 


9    ^ 


=  8 

•  *» 

is- 

a  '— 

o  ti  -^  ■• 
-H  *i  j<  g 

J5  '-I  a 

Z  u  -i  u 

»  »       m       ^  w 

WOPO  gh  k.Sp<  >^*1  u 


o? 


u 

M 

i 


r-aui^-ii        am—" 

-*9  7  |g 


-  i 


s   i 


S5 

in 

^^ 

'4  a  h 

■    •  *• 

O    3 

^J  o  9 
a  -H  Si 

a  n 

•     O    iJ 

**    •  C 


^11 


\ 

I 


..=  i  i 

a  5  —  T>  « 

**  6  X 

A  8  lO  * 

4J  4J  *«  >^  •*  -H 

*  >-  8  ^'<4 

o  o  j  tJ  a 


a 

Si 

u 

o 

a 
u 

s 


k 

a 
> 


-    5 


9  o« 


P  O    3    3 

M  0    0 

o 

M  O 

II  —         • 

o  »• 


o 
o 


5 

-4 
> 


-  s 


o 

•4 

5 

> 

u 


■o 

« 


3 
S 


5 

u 
o 

o 
u 

3 
t; 


*    -   I   k 
.    ^   ?   6 

8    ^    •. 


;:  I 


iJii5 


11 


5 

a 

b 

o 


8. 

• 

I. 


1 


S 
S 

Oi 

3 

i? 


1,       a      -■  - 


9 
•  3 
■     o 
a  X 


S5 


S3 


•I 

la 


3? 

a 

M  E  a 

>v 

sl^ 

*J5 

s.^ 

•'1 

-4  r^ 

-            ft     :o 

•4  _                      »  -            . 

%'\ 

J  n                   a  ^        r«  _. 

list 

w  —   -          ^  1       -a 
*  E  u          'Q  "       i. 

^8i5       1-    £ 

•Jii 

H 


& 

a 

U 
U 


41 

8 


3 


-I  c 


■"  *"  3 

.-4   3 

llll 

0  I.  a  • 
o  *J  -<  o 

-t    ■    U   i-l 

■H  S  u 


Sfv 


5 


II 


II    II 


U) 

s. 


«<•  " 


oo 

3  ^ 

3 


1. 
1 

■ 


I. 
0. 


■0  m  tJ 

"^  *      o  •8 

— .      II  II 

■o  — o  >.— . 
a       U  >. 

(t    II    *J        — ' 


S 

■D 

a 
JQ 

o 
•0 


u  / 


O 


o 

5fr 


^5 

IS 


■ 
II 

^-5 


•o  £  u 
o>  *I  c 
O  3  -3 


o  oo  «— 

,iS~ 


2    6| 


4> 
2 


i8 


i:3 

S3 

•  u 

8| 


o 


1 

I 

O 

u 


i 

>8i 


o  • 


o 

i 

t 


5 

> 


•  *1 

I       1 


4J 

O 


1 

1: 


a  -H 


«3 

g: 

II 

•H    3 

II 

t»  s 

a 


«  ■•  «     « 

^  S,  *  ^  to 

M  ?  •«-. 

1  ^  hi 

-  ^  U^ 

•H  -H  JS  u  .,1.. 

V  i  f  4  < 


i  o 

II 
II 


^1 


:f 


::   I 


^i1 


a       a 

•-4   >    -4 

a 
a  fH 

*  *  c 


6  *<       -H 


4j 

M  a 
■g  V  »4 

8  -4  M 
^5 


•  rt  II   —'■-" 
I  1  >.^ 


II 


S| 


?S' 


^ 


1 


!1 

1: 


1 


B 


>-l 

-J 


§1 


C/?'^>(  <"'' 


Fig,  1.   Arrangement  of  equipment  for  plane  of  light  triangulation. 
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Fig.  2.  Definition  of  quantities  used  in  the  derivation  (see  text). 
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Rg.  3.   Ratio  values  along  this  screen  define  </(p). 
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Fig.  4,.  If  measurements  are  made  whfa  projector  in  the  calibration 
location  some  regions  of  the  "workspace"  are  missing. 


Fig.  5.   Intensity  image  of  the  workspace  as  seen  by  the  video  camera. 
Note  the  dark  paper,  and  grain  on  the  wood  block. 


Fig.  6.   Depth  image  of  the  workspace.  Brighter  values  are  farther  from 
the  camera.   Shadows  are  tagged  black  (no  depth  information). 


Fig.  7.   Sequential  2  an  sections  through  the  depth  image.  AD  portions 
of  the  image  lying  outside  a  section  are  black. 
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